Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support reading events from Kafka #467

Merged
merged 4 commits into from
May 10, 2024
Merged

Support reading events from Kafka #467

merged 4 commits into from
May 10, 2024

Conversation

hmadison
Copy link
Contributor

What type of PR is this?

/kind feature

Any specific area of the project related to this PR?

/area plugins

What this PR does / why we need it:

This makes an attempt to resolve a line item in #228 around Kafka support. The changes adds a new plugin which allows for a consumer group to read the “bytes off the wire” from a given set of Kafka topics. It explicitly does not support parsing these bytes since Kafka’s internal model is inherently byte based. Instead, we can delegate the parsing to extractor plugins which can build on top of the events provided by this plugin.

Which issue(s) this PR fixes:

In part #228.

Special notes for your reviewer:

  • From a test perspective, this makes use of testcontainers/docker to validate the behavior of the plugin. I’m not sure if this work under the current test infrastructure for plugins.
  • This seems to have some event time sensitivity issues. At low levels of traffic messages are being read in the go routine but are dropped downstream.

@poiana
Copy link
Contributor

poiana commented Apr 17, 2024

Welcome @hmadison! It looks like this is your first PR to falcosecurity/plugins 🎉

@poiana poiana requested review from jasondellaluce and leogr April 17, 2024 15:14
@leogr
Copy link
Member

leogr commented Apr 18, 2024

Hey @hmadison

Welcome and thank you for this PR! :)

Just a couple of questions:

  • Could you like to register your plugin and pick the next available ID?
  • Are you willing to maintain this plugin in the long-term? If yes, I'd recommend proposing yourself as reviewer for this plugin.

cc @falcosecurity/plugins-maintainers

Also, I'm not a Kafka expert, so I'm not likely the right person to review this PR. Please, be patient we will find someone :)

Thank you 🙏

@hmadison
Copy link
Contributor Author

Hi @leogr,

I'm fine maintaining this long term and helping with review.

Thanks,
Hunter

@leogr
Copy link
Member

leogr commented Apr 23, 2024

Hi @leogr,

I'm fine maintaining this long term and helping with review.

Thanks, Hunter

This is great! So let's proceed this way:

  • We will review and merge this PR (it will take a bit, but we do our best). Cc @Issif @alacuku. May you help with this?
  • Once the PR is merged, you will add yourself as a reviewer in a separate PR (and I will sponsor you), then we will do the onboarding

Would this work for you? :)

leogr
leogr reviewed Apr 23, 2024
View reviewed changes
registry.yaml Show resolved Hide resolved
@Issif
Copy link
Member

Issif commented Apr 23, 2024

/assign

@Issif
Copy link
Member

Issif commented Apr 23, 2024

Hi,

I'll take a look when free, it can take a week or 2.

@hmadison
Copy link
Contributor Author

@leogr,

Works great!

Thanks,
Hunter

@leogr
Copy link
Member

leogr commented Apr 23, 2024

/assign

leogr
leogr reviewed May 2, 2024
View reviewed changes
plugins/kafka/OWNERS Outdated Show resolved Hide resolved
@alacuku
Copy link
Member

alacuku commented May 7, 2024

Also, this test is constantly failing, can you take a look? Screenshot from 2024-04-30 18-49-10

cc @alacuku @Andreagit97 @FedeDP. The failure might not be related to this PR. Could you please take a look at it, too? 🙏

It seems related to the go version:
go: go.mod requires go >= 1.22.2 (running go 1.21.9; GOTOOLCHAIN=local)

@hmadison, could you please bump the go version used in our CI?

@leogr leogr mentioned this pull request May 8, 2024
Merged
@github-actions GitHub Actions
Copy link

github-actions bot commented May 8, 2024

Rules files suggestions

@leogr
Copy link
Member

leogr commented May 8, 2024

Hey @hmadison

Unfortunately, we do not allow merge commits in our git history.

Could you rebase this PR and remove the merge commit, pls? 🙏

hmadison added 4 commits May 8, 2024 07:29
@hmadison
Update Golang build environment to 1.22
e2a9252 
Signed-off-by: Hunter Madison <[email protected]>
@hmadison
Support reading events from Kafka
28c480b 
Signed-off-by: Hunter Madison <[email protected]>
@hmadison
Register plugin as ID 18
8a526ca 
Signed-off-by: Hunter Madison <[email protected]>
@hmadison
Add a Readme and Owners
fd37dc6 
Signed-off-by: Hunter Madison <[email protected]>
@github-actions GitHub Actions
Copy link

github-actions bot commented May 8, 2024

Rules files suggestions

@leogr
Copy link
Member

leogr commented May 9, 2024

/verify-owners

@github-actions GitHub Actions
Copy link

github-actions bot commented May 9, 2024

Rules files suggestions

leogr
leogr approved these changes May 10, 2024
View reviewed changes
Copy link
Member

@leogr leogr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved!

🥳

Thank you!

@poiana poiana added the lgtm label May 10, 2024
@poiana
Copy link
Contributor

poiana commented May 10, 2024

LGTM label has been added.

Git tree hash: 31e71f09208110d64fbe2a612d02e6f4ea7ba19f

@poiana
Copy link
Contributor

poiana commented May 10, 2024

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: hmadison, leogr

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@leogr
Copy link
Member

leogr commented May 10, 2024

Also:

I'm going to release the plugin next week

@leogr
Copy link
Member

leogr commented May 10, 2024

/hold cancel

@poiana poiana merged commit d486cb0 into falcosecurity:main May 10, 2024
12 of 13 checks passed
@hmadison hmadison deleted the hm/kafka branch May 10, 2024 10:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Issif Issif Issif left review comments

@leogr leogr leogr approved these changes

@jasondellaluce jasondellaluce Awaiting requested review from jasondellaluce

Assignees

@leogr leogr

@Issif Issif

Labels
approved area/plugins dco-signoff: yes kind/feature New feature or request lgtm size/XXL
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@hmadison @poiana @leogr @Issif @alacuku